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1. INTRODUCTION 

Ambient intelligence aims to create environments with communication, computation and decision- 
making capacities. These features help to enhance the way in which users interact with various environments, 
based on real-time information gathered and historical data accumulated. Ambient intelligence is built around 
ubiquitous systems (known also as pervasive systems) by introducing a key-element: explicit requirement of 
intelligence. This way, mobile applications can sense their environment, understand the context of collected 
data and adapt their behaviour accordingly [1]. The context refers to information used to depict the situation 
of entities considered relevant to the interaction between a user and its environment. 

In this work, we consider the case of playing multimedia documents in ubiquitous systems. These 
documents include media objects of various natures such as text, image, audio, and video; web pages are a 
good example. This type of documents plays an important role in several fields such as healthcare, distance 
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learning and tourism. In pervasive environments, users context may affect the proper presentation of 
multimedia documents; we cite for instance: noise level, location, screen size, and battery level. One 
approach is to adapt their content so that they fit as much as possible to the current context. For example, if 
the user consults multimedia documents through a tablet at home, the system may suggest displaying videos 
contents through the smart TV since the screen is larger. 

In the literature, several approaches have been proposed in order to adapt the content of multimedia 
documents in context-aware pervasive environment [2]-[21]. These approaches are classified into four 
categories: server-side adaptation, client-side adaptation, proxy-based adaptation and peer-to-peer adaptation. 
Within the adaptation process, the user context is first sensed. Then, the constraints making the documents 
features non-compliant with the context are identified. Finally, adaptation actions are inferred and performed 
so as to provide adapted documents. Although these approaches show interesting results, they, however, deal 
only with near real/real-time data collected to adapt the documents in accordance with context changes. In 
other words, most of them do not handle historical users data (HUD) accumulated within the adaptation 
process. Indeed, HUD help to improve the overall system behaviour; we mention for instance: applying 
machine-learning techniques, making adaptation rules learning, and performing data mining processes. 
Moreover, most of multimedia document adaptation (MDA) processes focus on executing the adaptation 
module in a single location (clients, proxy or server side), in particular at servers. In several cases, such a 
choice hinders the efficient execution of MDA processes within context-aware pervasive systems that 
involve properties such as proactivity, mobility, cross-platform, self-tuning, and adaptation. 

Starting from this context, we propose a software component to manage the HUD resulting from the 
execution of MDA processes for further processing. The proposal allows the storage, retrieval and analysis of 
historical data as it stores context values and corresponding adaptation actions. To this end, we rely on a 
flexible and agile architecture that can easily change form (polymorphism) to adapt to different categories of 
MDA processes, to users preferences (e.g. data sharing and privacy, and personalized processes) and to 
context requirements (e.g. computation resources). The aim is to provide MDA processes with more options 
to efficiently process HUD in different locations (clients, proxy and servers sides), depending on different 
users and context factors. For instance, let consider two users endowed with devices of different capacities: a 
laptop user and a smartphone user. If the laptop user does not want to share his historical data, it is then better 
to store and analyze these data through his device. In contrast, if the smartphone user wishes to share his 
historical data with other users, it is then better to store and analyze these data via a server. 

Depending on the category of the adaptation process, three variants of HUD management are 
proposed in addition to their hybridization: client-side management, proxy-based management, and server- 
side management. To that end, we first model the context as well as the required adaptation actions using the 
oriented-object approach. Then, we transform the resulting modelling into relational and NoSQL schemes 
using conversion procedures. Finally, algorithms for storing, retrieving, and analyzing data are designed. Our 
proposal is validated through a set of experiments considering scenarios implemented in a real prototype for 
which the HUD are stored in different ways. In particular, we emphasize the usefulness of our proposal by 
treating use cases on context-aware adaptation rules learning. 

The rest of this paper is organized as follows. Section 2 provides the basics of MDA processes and 
discusses some related works. Section 3 presents the proposed approach for managing HUD resulting from 
execution of MDA processes. Section 4 summarizes the experimental part and discusses the obtained 
numerical results. Finally, the paper ends with some concluding remarks and future directions in section 5. 


2. THE COMPREHENSIVE THEORETICAL BASIS 
2.1. Multimedia documents adaptation in ubiquitous systems 

As the context in pervasive systems is continuously changing over time, some constraints and 
difficulties may be encountered due to multimedia documents features that conflict with the current context. 

In this respect, several adaptation approaches have already been proposed, in order to provide documents 

adapted to contextual situations [2]-[21]. These proposals differ from each other mainly in the way the 

adaptation process is carried out as well as in the documents types treated. As discussed in [1], context-aware 
pervasive systems are built around three-layered elements: sensing, thinking and acting. Next, we give details 

about these elements in the case of MDA processes, as represented in Figure 1. 

a. Sensing layer: this layer includes a single step namely context collection. It aims at updating the elements of the 
context by acquiring data from the physical world using different sources such as sensors. The context includes 
different types of information that can be classified into physical context (e.g. lighting, and noise level), user- 
related context (e.g. profiles, and locations), computational context (e.g. network connectivity, and 
communication bandwidth) and temporal context (e.g. season, year, month, day, and time of day) [1]. 
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Generally, these data are obtained either by events triggered by context changes or by time driven methods 


according to which the context values are calculated during periodic checks. 


b. Thinking layer: this layer comprises three steps namely context-modeling, context-reasoning, and decision- 


making which act as follows: 


- Context-modeling: the context information collected is raw because it only includes values measured by 


various sources (low-level context data); hence, it should be analysed further to take desired actions. To this 
end, the gathered information is organized according to an abstract context model such as key-value 
models, markup scheme models, graphical models, object-oriented models, logic-based models, and 
ontology-based models [22] (high-level context data). The goal is to allow processing context information 
at a higher level that helps to reason upon it. 

Context-reasoning: the reasoning step involves the high-level context data to identify the constraints and 
conflicts that hinder playing the documents properly; it usually takes place in two forms: real-time 
reasoning and on-demand reasoning. Real-time reasoning refers to event driven and time driven context; 
hence, it is continuously executed in order to infer the required adaptation actions. On-demand reasoning is 
triggered by a request from a user or application for many purposes such as resources discovery (e.g. 
surrounding users/objects) and recommendations (e.g. suggestions regarding environmental conditions 
such as lighting). In the end of this step, a set of adaptation actions are inferred in order to provide adapted 
documents that fit as much as possible the contextual situation, often through "if ... then" rules. 
Decision-making: the objective of this step is twofold: the determination of the adaptation actions to be 
executed and the generation of the adaptation plan. The decision-making step deals with the adaptation 
actions produced in the context-reasoning step in three possible modes: i) the automatic mode according to 
which the system keeps these actions without involving the interaction with the user, ii) the semi-automatic 
mode according to which the system proposes to the user the inferred actions, in order to choose the most 
suitable among them, and iii) the manual mode according to which the adaptation actions are entirely 
generated by a request from the user based on specific needs. 

Once the set of adaptation actions has been determined, the adaptation plan is generated by binding each 
action to a set of possible execution paths, depending on the available candidate adaptation services. Then, 
the best paths are selected based on a service selection procedure according to QoS parameters (e.g. 
response time). 


c. Acting layer: the last step consists in performing the adaption plan by executing the selected adaptation 
services, in order to meet the requirements of the environment and the user. These services are applied to media 
objects composing the documents to produce a set of adapted objects. These contents are then arranged in an 


adapted document so that it is displayed in a suitable form to the current contextual situation. 


Let us consider a mobile user consulting online course. For the sake of simplicity, we assume that 
the context elements include the location, current activity and preferred language of the user. Table | gives 


some typical context values, conflicts, and adaptation actions related to the MDA process used. 


Multimedia 
document 
(input) Environment Devices features 
| so. Om 
d 20r y @e 
8 & User profile 
«@ om (preferences _.) 


NI 4&4 


Adapted 
document Context reasoning 
(output) On-demand 


Figure 1. The general architecture of a standard MDA process 


Table 1. Some typical context elements, conflicts and adaptation actions 


Context element Value Conflict Adaptation action 
Location Bus Auditory contents should not be played Exclude audio 
Language Spanish The content cannot be understood Language translation 
User activity Driving Hands and eyes cannot be used properly Voice-commands 


2.2. Involving historical users data in context-aware multimedia documents adaptation processes 


In the litterature, much research has been conducted on context-aware maultimeida adaptation 
[2]-[21]. Depending on where decision-making and adaptation actions take place, MDA processes can be 
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divided into four broad categories [23]. The adoption of a particular category depends on many factors such 
as devices computing features, the communication bandwidth, and the available resources. 

a. Client-side adaptation: the devices running the documents make the adaptation process by themselves 
(e.g. [17]). This option allows more responsiveness; however, the limited resources of some devices (e.g. 
battery, CPU) may be a hindrance to the execution of adaptation actions. 

b. Server-side adaptation: the devices playing the documents send adaptation requests to a server that deals 
with them (e.g. [20]). Although this option frees the client side from a tedious process that could slow 
down performance, this may affect the server response time due to overload. 

c. Proxy-based adaptation: a proxy is placed between the client and the server in order to act as a mediator 
while running the adaptation process (e.g. [7]). The proxy frees both client and server from intensive tasks 
by leveraging its computing power; however, this option may involve too much communication for 
negotiating which adaptation actions to execute. 

d. Peer-to-peer adaptation: the devices playing the documents can communicate with multiple platforms, 
including other devices, to perform the adaptation process (e.g. [5]). This option takes advantages of the 
peer-to-peer paradigm to ensure a balanced load while executing the adaptation actions. Despite this, the 
process efficacy depends on the number of connected nodes. Furthermore, the network administration is 
difficult due to its decentralized nature. 

Table 2 provides a comparative summary of some of the MDA approaches cited above. The aim is 
to compare them according to their main limitations concerning the way the reasoning phase is performed. 

Such a comparison helps propose improvements and contributions with respect to related research-works. 


Table 2. A comparative summary of some MDA approaches 


Approach Category Overview Main limitation 
Hai et al. Peer-to-peer A service-based approach for MDA in which adaptation services No support of semantic aspects in 
(2012) [5] are available locally and remotely on multiple platforms. profiles and adaptation services. 


Dromzée et 

al. (2013) [7] 
Bettou and 
Boufaida 

(2017) [11 
Adel et al. 
(2017) [13 


Saighi et al. 
(2017) [12 


Khallouki 
and Bahaj 
(2017) [14 
El Guabassi 
et al. (2018) 
[16] 
Belhadad et 
al. (2018) 
[17] 

Saighi et al. 
(2020) [20] 


Proxy-based 


Server-side 


Server-side 


Server-side 


Server-side 


Server-side 


Client-side 


Server-side 


A service-based context modeling approach for MDA that links 
the context information using semantic generic profiles. 

An MDA approach based on formal concepts related to 
technical adaptations to select relevant policies by detecting the 
conflicts between documents properties and users preferences. 
An MDA approach that involves social impact inferred from 
communities on Twitter and Facebook, to make an effective 
composition of adaptation services using semantic relationships. 
A handicap-based MDA that exploits an ontology to provide 
context-aware assistance by binding each context constraint to a 
physical handicap and thus to an adaptation action. 

A context-aware MDA approach based on cooperative multi- 
agents systems that uses semantic web services to predict users 
context and provide adapted contents accordingly. 

An XML-based personalization of course content in ubiquitous 
learning, considering learning styles and context-awareness 


An XML-based multimedia specification that describes the 
structure of web contents according to users context and profile. 
The content is adapted by modifying their spatial structures. 

A graph-based MDA that allows the inference of multiple 
adaptation actions using semantic reasoning upon users context. 


No support of 
mechanisms. 

No support of semantic aspects to 
deal with context constraints. 


reasoning 


No support of the compatibility 
between the inferred adaptation 
actions. 

Only one adaptation action is 
inferred per adaptation request (no 
support of multiple actions). 

No prototype was implemented 
for the proposal to be tested and 
evaluated. 

No support of semantic aspects to 
deal with context elements. 


Only the spatial dimension of 
objects is handled but not their 
temporal synchronization. 

No parallel computing support for 
reasoning (performances issues). 


By analyzing several MDA approaches [2]-[21] (see Table 2), we could check that they focus 


mainly on context collection, representation, and interpretation. In particular, the reasoning mechanisms only 
rely on profiles and current context values in such a way that the adaptation task is performed in a single 
location (i.e. client, proxy or server sides). Thus, the storage and analysis of past context values and 
corresponding adaptation actions are not supported in order to be involved within adaptation processes. 
Logging of history information is supported by many context-aware pervasive applications as it helps to 
provide prior knowledge about users decisions so that they can be taken into account in future processing. 
We cite as examples machine-learning approaches for: IoT cultural data [24], context-aware intrusion 
detection [25], smartphone data analytics [26]-[29], activity recognition [30], and healthcare support [31]. 
Likewise, HUD can contribute in improving the overall behaviour of MDA processes by performing 
several advantageous tasks such as machine-learning techniques, statistical methods for building 
recommendation systems, and adaptation rules personalization. In this case, two types of HUD can be 
considered either jointly or separately. The first type concerns the multimedia documents consulted by 
emphasizing on general information about users and resources accessed. The second type is related to 
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specific features of ubiquitous systems (e.g. sensors, and human-computer interactions.) insofar as the 

adaptation process should deal with context values and adaptation actions. Regarding the first type of HUD, 

several researches in other fields have already been conducted in this respect; web log mining is a good 

example [32]. In contrast, there is still a general dearth of MDA approaches treating the second type of HUD. 

Indeed, even though there are some MDA-based applications that involves certain HUD (e.g. [33], [34]), they 

however do not provide effective mechanisms to manage such data since they only focus on analyzing them 

either through datasets or by certain collected data. Generally speaking, log data are extremely diverse and 

processing them is unfortunately quite complex. This is due to the lack of standards and agreements outlining 

the granularity, structure, content, format, and level of details provided by log events [35]. This raises certain 

issues related to their management, which are further addressed later in this paper: 

- The first concern is about the determination of which data will be stored; this is a crucial question that 
depends on the intended purposes of their storage. 

- The second concern is about the format in which the data will be stored. This aspect has a direct link with 
efficient data storage and retrieval for further processing. 

- The third concern is about the place where the data will be stored. This is a very important aspect since it 
may lead to dealing with large volumes of data (most likely big data). 

- The last concern is about the way in which the data stored will be leveraged in useful tasks by integrating 
users experiences into MDA processes, in order to improve their effectiveness. 

Starting from this context, we propose to design a software component that handles HUD resulting 
from the execution of MDA processes for further processing. Compared to other MDA-based approaches, 
our proposal should be generic insofar as it can be integrated into different MDA processes categories in such 
a way that it allows the storage, retrieval and analysis of historical data as log data resulting from the 
accumulation of the decisions made by users. This way, the proposed component will provide MDA 
processes with more options to efficiently process HUD in different locations namely clients, proxy, and 
servers sides. To do so, we rely on a well-devised, flexible, and agile architecture that can easily change form 
(polymorphism), and adapt to context changes (e.g. computation resources), and users 
preferences/requirements (e.g. personalized processes, data privacy, and sharing). 


3. METHOD 
Our proposal consists of a software component for managing HUD resulting from the execution of 

MDA processes. The aim is to allow their storage, retrieval and analysis so as to involve them in users 

decision-making according to different ways. This is a complementary element to existing adaptation 

approaches that attempts to improve their overall performances since in most of them: 

a. Decision-making is performed by inferring the adaptation rules using only on-demand and/or real-time 
reasoning; thus, they do not imply HUD when performing the adaptation task. Our proposal allows MDA 
processes to leverage from HUD to perform several advantages tasks (e.g. machine learning tasks), in 
order to improve the adaptation process quality. 

b. The whole adaptation task is carried out in a single location (clients, proxy or server sides), in particular 
the client-server paradigm which is broadly adopted due to implementation simplicity and computation 
resources power. Even so, the efficient use of other paradigms can also be a suitable choice to deal with 
several difficulties (e.g. workload balancing). One way to take into account this issue is to rely on an 
effective hybridizations between different adaptation categories such that the MDA process is executed in 
a decentralized way by switching from one adaptation category to another in the event that the current 
solution shows its limitation. In this regard, our proposal is built around a well-devised, flexible and agile 
architecture that offers the possibility to treat HUD in different locations and in different ways, which: 

- makes the HUD manager potentially supported by all MDA categories. 

- provides more options for MDA approaches, depending on the current computation resources and 
users preferences (e.g. data privacy and data sharing), and 

- allows HUD to be managed by different devices (e.g. smartphones, laptops, and servers). 


3.1. The general architecture of the proposed HUD manager 
Now, we give details of the proposed HUD manager. Figure 2 shows its architecture and the 
potential interactions with MDA processes. The proposal comprises four components as follows: 
a. The database: it represents the storage medium for storing the HUD. 
b. The data logger: it makes systematic recordings of HUD in the database each time new data is acquired 
by the system. 
The data retriever: it allows retrieving HUD from the database to be used by the data analyser. 
d. The data analyser: it mainly consists of a bag of data analysis algorithms dedicated to different purposes 
such as machine learning techniques, and data mining methods. 


Q 
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Figure 2. Detailed architecture of the proposed component and its potential interactions with MDA processes 


The interaction between the MDA processes explained in subsection 2.1 and the HUD manager 
takes place through an invocation of the data logger and the data analyser, performed by the thinking layer, as 
shown in Figure 2. This layer includes the context-reasoning and the decision-making modules that constitute 
the core of the adaptation system and therefore the source of its intelligence. On the one hand, the decision- 
making module invokes the data logger to store HUD about the context values and the decisions made by 
users (step 1 in Figure 2). On the other hand, the context-reasoning module takes advantages of the 
functionalities provided by the data analyser that contribute in making future decisions (step 2 in Figure 2). 
Next, we summarize the three steps executed by the proposed HUD manager. 

- Step 1: HUD logging 

Once the decision-making module has generated the adaptation plan, it calls the data logger 
component to store information about the corresponding context values and adaptation actions made by the 
application. In turn, the data logger accesses the DB through a query language-—that depends on the 
implementation—in order to insert these data as new entries (see Figure 2). 

- Step 2: data analysis 

The data analyzer component provides a bag of data analysis algorithms that can be used by the 
context-reasoning module, in order to improve its effectiveness. To do so, it calls the data retriever to acquire 
data for analysis according to two methods: real-time analysis and on-demand analysis. Real-time analysis 
refers to event driven methods triggered whenever new entries are inserted into the DB; it is continuously 
executed in order to update the context-reasoning module as needed, depending on the purposes of the 
application. Algorithms in this category should be lightweight in order to maintain the responsiveness of the 
system with respect to application usage. A typical example of this category is the statistical analysis 
methods based on pattern frequencies for which the algorithm needs only to maintain the frequencies using a 
dynamic data structure. On-demand reasoning is triggered by a request from a user or application for many 
purposes such as machine-learning algorithms and recommendation systems. Algorithms in this category can 
be heavyweight. Moreover, they can be executed online or offline as they do not influence the responsiveness 
of the system with respect to application usage. 

- Step 3: data retriever 

The data retriever is designed according to the data access object (DAO) pattern that provides an 
abstraction between several database types as well as other persistence mechanisms and the other 
applications parts. This way, calls to the persistence layer goes through the DAO so as data operations are 
performed without revealing database details. Thus, the data retriever consists of a set of callable methods 
that retrieve data from the DB according to different input parameters. The implementation of such methods 
is based on query languages (e.g. SQL, and XPath) depending on the type of DB adopted. Two types of data 
retrieve methods are proposed: data retrieve based on triggers and data retrieve based on query languages. 
The first type of data retrieve uses triggered callable methods that are executed when certain conditions 
related to DB updates are met. The second type of data retrieve is based on parameterized callable methods 
that access and retrieve data from the DB according to the input parameters. 

The proposed HUD manager has several functions. This provides various options that help in 
efficiently designing MDA processes according to users requirements and context specifications. Depending 
on where the HUD is stored and processed, three methods in addition to their hybridization are proposed: 

a. Client-side management: the client devices store, retrieve and analyse HUD by themselves. 
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b. Server-side management: the HUD manager is installed in a server that stores, retrieves and analyses the 
data. The devices playing the documents (i.e. clients) send requests by invoking the data logger, and the 
data analyser as needed. 

c. Proxy-based management: the HUD manager is implemented in a proxy that is placed between the client 
and the server. 

d. Hybrid management: the components of the HUD manager namely the database, the data logger, the data 
retriever and the data analyser, are separately placed in the client device, proxy or server, depending on 
the current requirements of the MDA process. 


3.2. Implementation of the proposed approach 

To show the feasibility of our proposal, we develop a real prototype through which multimedia 
documents are adapted according to contextual situations while storing the HUD. In fact, from a functional 
point of view, most MDA processes work very well. However, when it comes to validating a given 
adaptation approach, the prototypes developed are often based on simulation to collect context elements; 
mainly for the elements related to connected objects (see for example [11], [12], [20]). Indeed, even if the 
simulation paradigm is very useful in many situations, the realization of a real prototype remains the most 
ideal, realistic and viable solution, in particular to deal with technical, and implementation issues. 


3.2.1. Context senesing 

Context values come from different hardware and software sources. Initially, they are collected as 
raw data since they only include measured values. These values serve as input data for the reasoning phase, 
the purpose of which is to build adaptation rules. Table 3 details the context elements used in our prototype. 


Table 3. Different context classes features 


Context class Attribute Domain of values Source 
Physical Noise Value = 0 Sensors 
Luminosity Value = 0 
Computational Battery level Level € [0, 100%] Device 
CPU load Load € [0, 100%] 
Memory usage Usage € [0, 100%] 
Screen size (width, height) (inch) 
Smart TV Availability € {0, 1} 
Network Bandwidth (value = 0) 
Type E {WI-FI, mobile data, ...} 
User Preferred language List of languages Profile 
Agenda List of timed tasks / places 
Location Lis of places {home, lab, public place, meeting room, ...} Profile/sensors/device 
Temporal Time (Date, time) Device 


3.2.2. Reasoning upon the context 

The reasoning upon context values takes place in two phases: qualification of context values and 
inference of adaptation actions. The first phase aims to qualify the quantitative values of the context so that 
they are effectively used during adaptation actions inference and HUD analysis. Table 4 shows typical 
examples of context values quantification; these values are usually defined empirically. 

Once the context values are qualified, the second phase is to infer the adaptation actions. Each 
adaptation action consists of a rule taking the form of "if (conflicts) then (actions)". Table 5 gives details of a 
subset of typical adaptation actions. To build these rules, we are inspired by those used in [12], [20]. 


3.2.3. Execution of the adaptation actions 

Our prototype supports web contents adaptation according to user contexts without depending on a 
specific domain of application. The aim is to ensure the genericity of the prototype so that it can be adapted to a 
wide range of applications using multimedia documents with simple objects (e.g. texts, images, and videos) or 
those composed of a combination of different types of contents. Generally, there exist several types of document 
sharing, depending on applications scopes; we cite as examples file transfer protocol (FTP), peer-to-peer sharing 
and cloud service sharing (e.g. [36], [37]). Regarding our prototype, the HTTP protocol was adopted to share a 
set of multimedia documents hosted on a server, due to simplicity of implementation. 

The adaptation of multimedia documents takes place by involving a set of adaptation actions. Each 
adaptation action is performed by executing one or several adaptation services, depending on the nature of 
the media objects in the documents. The description of the adaptation actions given in Table 5 is as follows: 
a. Speech to text: it mutes the sounds of auditory objects by replacing them with textual content (sounds on 

videos are muted and subtitled while audio contents are converted to text). 
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Contrast adjustment: it highlights visual objects contrast according to the brightness ratio. 

Video to images: it converts videos to images sequences. 

d. Language translation: this action translates media objects written in a source language to the target 
language specified in the users profile. 

e. Format change: it changes the format of a media object to another format (e.g. mp3 to mp4). 

Video in smart TV: this action plays video objects in a smart TV since the screen size is larger. 

Text summarizing: this action summarizes the content of text objects. 


o g 


gq m 


Table 4. Qualification of quantitative values of typical context elements 


Attribute Quantitative value v Qualitative value 

Noise V < Vmin Quiet 

VE [Vmin Vmax] Less noisy 

V > Vmax Noisy 
Luminosity V < Vmin Penumbra 

V €E [Vining Vmax] Luminous 

V > Vmax Very luminous 
Battery level v < Vmin Low 

VE [Vining Vmax] Medium 

VS Vinay Full 


Table 5. Some typical rules for carrying out adaptation actions 


Rule ID Conflicts Adaptation actions 
R1 if (public place or noisy environment) then Speech to text 
R2 if (very luminous environment) then Contrast adjustment 
R3 if (low battery or overloaded memory) then Speech to text 
Video to images 
R4 if (inappropriate language) then Language translation 
R5 if (overloaded CPU) then Format change 
Video to images 
R6 if (smart TV available and small screen) then Video in smart TV 
R7 if (busy agenda) then Summarize text 
R8 if (communication bandwidth is bad) then Format change 


3.2.4. Hardware and software configuration 

The hardware configuration includes an arduino UNO board on which three sensors are connected: 
sound, GPS, and photoresistor light sensors. Our prototype uses WiFi and bluetooth low energy (BLE) 
connectivity. On the one hand, WiFi is one of the most popular wireless communication protocols. In 
addition, it is very compatible with a wide range of devices. On the other hand, BLE-based devices are very 
adopted in IoT implementations since they provide low power consumption and consistent connectivity. 
These protocols are used within the sensing layer shown in Figure 1, to collect the context elements from 
various sources. They are also used in the intercommunication between wired and wireless devices. Our 
prototype can easily be expanded so that it supports other communication protocols such as zigbee, in order 
to satisfy the potential demands for compatibility with installations based on other ubiquitous IoT protocols. 

The hardware configuration also includes two HP Z640 stations (xeon CPU composed of 20 cores 
and 64 GB of RAM) used as a proxy and a server, and a smartphone (samsung galaxy A10s with helio P22 
CPU and 2 GB of RAM) used as a client device on which the documents are run. Adaptation services calls 
are performed using the following java APIs: google cloud speech API (details can be found on the website: 
https://cloud.google.com/speech-to-text/), yandex translate (available at: https://tech.yandex.com/translate/), 
fast forward moving pictures expert group (FFMPEG) (downloadable at: https://www.ffmpeg.org/) and Jave 
(downloadable at: http://www.sauronsoftware.it/projects/jave/). 

The database component is implemented using relational and NoSQL databases as they operate 
independently of any specific data analysis framework; the aim is to compare the two models. On the one 
hand, relational databases allow handling important volumes of structured data while offering efficient 
functionalities for data storage and retrieval. On the other hand, NoSQL databases allow handling large 
volumes of data at high speed with scalable architectures according to different degrees of data 
structuredness. Besides, there have been significant efforts made to build bridges between oriented-object, 
ontology and key-value paradigms on one side, and relational and NoSQL databases on the other side (e.g. 
[38], [39]). The data to be stored are restricted to qualitative values of the context elements and the 
corresponding adaptation actions resulting from changes in context over time. The static aspect of the HUD 
structure is specified by the class diagram shown in Figure 3. 
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ContextSituation 
User Physical Computational +id 

+user_id +noise +battery_level 

+location +luminosity +CPU_load Context 

+preferred_language +memory_usage +context_id 

+agenda +screen_size ~* 0.7 
compe +smartTV Action 
+time +network_type vaction id 
+date +network_bandwidth g 


+action_name 


Figure 3. The static aspect of the abstract structure of the HUD 


Depending on where the HUD are stored, several database variants are implemented (see Table 6). 
Concerning the relational databases, we have used SQLite databases for client-side methods and MySQL 
databases for both proxy-based and server-side methods. Regarding NoSQL databases, we have used JSON 
files for all methods, in particular JSON files in hadoop distributed file system for sever-side approaches. 

Regarding the implementation of the remaining components namely the data logger, retriever and 
analyser, we rely on the oriented-object paradigm features. More specifically, we use Java interfaces since 
they help to design processes specifications that can be implemented in different ways. Table 7 shows the 
implementation details according to the different data management methods. 


Table 6. Technical choices to implement the database component in the HUD manager 


Management method Relational model NoSQL model 
Client-side SQLite database JSON file 
Proxy-based MySQL database JSON file 
Server-side MySQL database JSON file in hadoop distributed file system 


Table 7. Technical choices to implement the remaining components in the HUD manager 


Management method Technical choice for implementation 
Client-side Java language for Android systems 
Proxy-based Java language for standard systems 
Server-side Java language for systems based on map/reduce paradigm 


4. RESULTS AND DISCUSSION 

To validate our proposal, three tests are conducted. The first test consists in a case study through 
which we show how the data are organized in the database as well as a typical example of the adaptation 
result of a multimedia document. The second test aims to measure and evaluate the performance of our 
proposal in terms of processing time and size of data required to process the HUD. Finally, the last test 
consists in a case study through which we show the usefulness of storing the HUD by considering a typical 
example of how HUD contribute to the personalization of adaptation rules. 


4.1. Formating and storing the HUD 

This section shows how HUD are formatted and stored in the database. Intuitively, there are two 
methods to transform the HUD modeled in Figure 3 into relational model and JSON objects. In the first case, 
the log data take a columnar form by grouping together all attributes of HUD classes; thus, the data are 
structured as a set of entries. This format is useful for applying data analysis techniques. In the second case, 
the log data take the form of structured entities (linked tables, and nested objects) using mapping procedures 
while maintaining the relationships among them (e.g. [38], [39]). This format is more suitable for data 
readability at a high specification level. Regarding our work, we adopt the former case since our main goal is 
to perform data analysis. Below, an illustrative scenario for which the HUD are stored by executing an 
adaptation process triggered under certain assumptions. For this purpose, we rely on the prototype developed 
by running a client-side adaptation system that deals with web pages contents according to the context 
situations of users while handling their HUD. We note that the approach is general so that it can easily be 
adapted to several dedicated MDA processes such as medical applications, ubiquitous learning, and tourism 
filed. Thus, the scenarios considered only represent demonstrations by way of non-limiting examples. 

Mr. John is a medical-researcher at university; he generally connects to medical forums for 
discussion. Given the nature of his job, John is often on the move for consultations, and conferences. When 
traveling, he wants to consult a multimedia document related to radiology. The document is written in french; 
it consists of text, audio, image, and video contents, as illustrated in Figure 4. 
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Définition Illustration Vidéo-fracture 


La radiologie et l'imagerie médicale 
est un auxiliaire des autres 
spécialités médicales. Cet outil 
majeur d'investigation intervient 
dans le dépistage (grâce à une 
mammographie par ex.) le 
traitement et le suivi de nombreuses 
maladies pathologies 


Avis du chirurgien 


-e o00  —e 


Figure 4. Structure of the original document 


Let assume that John is in a conference room and uses his tablet to access the document. The battery 
level of his tablet indicates that it has reached 25%. The contextual constraints inferred by the adaptation 
system are as follows: public place, low battery and inappropriate language. Hence, rules R;=ı, 3,4 in Table 5 
are executed to provide an adapted document, as shown in Figure 5. On another side, the system invokes the 
data logger to store the log data in the database, according to the methods shown in section 3. A typical HUD 
input content is encoded as in the following JSON code. 


{"ContextSituation": {"situation_id": "1", "context_id": "5", "user_id": "1", "preferred_languge": "inappropriate language", 
"location": "public place", "agenda": "busy", "battery_level": "low", "CPU_load": "low", "memory_usage": "low", 


mon won mom 


"screen_size": "small", "network_type": "mobile data", "network_bandwidth": "good", "smartTV": "unavailable", "noise": "quiet", 
"luminosity": "luminous", "time": "9:33:54", "video_in_TV": "false", "language_translate": "true", "txt_summarize": "false", 


mon 


"speech2txt": "true", "contrast_adjust": "false", "format_change": "false", ''video2image"': "true" } 


Definition Illustration Video-fracture 


Radiology and medical imaging is a 
valuable auxiliary to other medical 
specialties. This major investigation 
tool intervenes in the screening 
(thanks to a mammogram for 
example) the treatment and follow- 
up of very many pathologies or 
even in emergency 


Surgeon’s opinion 


Severe trauma usually leads to injuries rib cage wall. For example of reaps a fracture you 
can make the pain associated with this injury is breathing very difficult. 


Figure 5. Structure of the resulting adapted document 


4.2. Performance analysis 
4.2.1. Running time and data size measurements 

Next, we evaluate the different database models shown in Table 6. The evaluation of performance is 
carried out by measuring the running time and the data size required for processing the HUD according to the 
current number of entries in the database. In particular, we study the average time for inserting one HUD 
entry in the database, the required time for loading the database content in memory to make data analysis and 
the storage space required for storing the HUD. The tests are performed using the hardware configuration 

mentioned in subsection 3.2.4. Figure 6 shows the obtained results by assuming that each user stores a 

maximum number of entries=10000. The users number is set to 1 for client-side HUD management and to 50 

for proxy-based/server-side HUD management. Figure 6 includes: 

- Figures 6(a) and (b) to plot the variations of the average time required for inserting a new entry in databases. 

- Figures 6(c) and (d) to plot the variations of the average time required for retrieving data entries from 
databases. 

- Finally, Figures 6(e) and (f) to plot the variations of the size of the data stored in databases. 

By analyzing the curves illustrated in Figure 6, one observes that for both cases (i.e. client-side, and 
proxy-based/server-side HUD management): 

- The time required for storing one entry is quasi-constant regardless of databases sizes (see 
Figures 6(a) and (b)). In particular, NoSQL databases are well performing for data storage compared to 
relational databases. This is because relational database management systems use advanced techniques to 
optimize the data storage, which affects the processing time. 
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Figure 6. Statistics about running time and data size required for processing HUD: (a) time for inserting one 

entry in client-side methods, (b) time for inserting one entry in server/proxy methods, (c) time for retrieving 

data in client-side methods, (d) time for retrieving data in server/proxy methods, (e) data size in client-side 
methods, and (f) data size in server/proxy methods 
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- The time required for retrieving and loading data increases linearly in accordance with the growth of 
databases sizes (see Figures 6(c) and (d)). This is expected as the volume of processed data becomes 
larger. Nevertheless, relational databases are well performing for data retrieve compared to NoSQL ones 
since the latter parse data before storing them as objects in memory. 

- The required space for storing data increases linearly in accordance with the growth of databases sizes 
(see Figures 6(e) and (f)). Besides, relational databases require less storage space given that NoSQL 
databases store entries as combinations of keys (i.e. objects and attributes), and values. To overcome this 
issue, it is possible to make the descriptors length as shorter as possible. Even so, NoSQL databases may 
excel since they do not require dedicated configurations either for exporting data or for accessing them in 
addition to their simplicity of implementation. 

Globally, the general performances of the proposed HUD management methods depend mainly on 
the available computational resources. For instance, during the experiments, we could check that the client- 
side method could not store more than 55.000 in SQLite databases. It also could not load more than 150.000 
entry in memory from NoSQL databases. Likewise, the proxy-based method could not load more than 
800.000 entry in memory. This leads us to care about issues related to efficiency of data storage, retrieval, 
analysis, scalability and others. 


4.2.2. Comparison of the proposed HUD management methods 

Now, we compare the proposed HUD management methods detailed in subsection 3.1 with each 
other based on systems ability to meet the following criteria: data volume, data performance, data retrieval, 
data analysis, scalability, data sharing, and data security. These criteria depend on certain factors that 
influence the general performances of the HUD management system such as data storage location, data size, 
computation resources, and number of users. Table 8 provides details about each criterion used for 
comparison purpose. 


Table 8. Criteria for comparing the proposed HUD management methods 


Criterion Overview Influencing Qualitative evaluation values 
factors High value Mediocre value Low value 

Data volume Data volume the system can process Computation Big data Large Small 
(storage and retrieval). resources 

Data Performance for storing/retrieving Number of Excellent Good Average 

performance large volumes of data. users 

Data retrieval Ability to load large volumes of data Data size Excellent Average Limited 
for processing. 

Data analysis Computational ability to perform Easy Feasible Difficult 
analysis tasks upon HUD. 

Scalability Number of users that the system can Scalable Less scalable Not 
handle. scalable 

Data sharing Ability to share HUD among users to Number of Suitable Less suitable Unsuitable 
enhance experiences. users 

Data security Ability to deal with privacy and HUD storage Strong Weak Very weak 
security of users data. location 


Table 9 summarizes the comparison between the proposed HUD management methods according to 
the criteria presented in Table 8. The statistics given in Table 9 help to determine the way in which the HUD 
can be effectively managed, and processed. Globally, server-side methods may excel because of the 
computation resources they have, the number of clients they can handle and the data volume they can store. 
In contrast, client-side methods show high data security since they store them on clients devices; thus, any 
access to data goes through client permission. 

Even so, these features may be affected by several factors such as the available computation 
resources, the current workload, load balancing issues, fault tolerance, and service availability. In such cases, 
hybrid management methods (i.e. those combining client-side, server-side, proxy-based, and peer-to-peer 
paradigms) help overcome these issues through an efficient placement of the HUD manager components in 
different locations, depending on the needs and requirements of both applications and users. 


Table 9. Comparison between the proposed HUD management methods 


Method Security Volume Performance Retrieval Analysis Sharing Scalability 
Client-side Strong Small Average Limited Difficult Unsuitable Not scalable 
Proxy-based Weak Large Good Average Feasible Less suitable Less scalable 
Server-side Weak Big data Excellent Excellent Easy Suitable Scalable 
Peer-to-peer Weak Small Average Limited Difficult Less suitable __Less scalable 
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4.2.3. Comparison of our proposal over other approaches exploiting HUD 
Finally, we compare our proposal with three research-works [33], [34], [40] that exploits HUD in 

pervasive environments (see Table 10). The purpose is to show the strengths of our proposed approach as 

well as the value added so as to improve MDA processes. To do so, we first consider the following points: 

- Approach scope: it describes the scope of a given research-work. 

- Approach category: it refers to the HUD management methods discussed in section 3, namely server-side, 
proxy-based, and client-side. 

- Efficient historical data management: it refers to whether a given approach uses/provides efficient 
mechanisms to handle the HUD. 

- Technical details: it refers to whether a given research-work provides technical details about HUD 
handling. 

- Historical data use: it refers to the purpose for which the HUD is leveraged in useful tasks. 

- Historical data involved: it refers to the elements composing the HUD. 

- Approach flexibility: it refers to whether a given approach is agile and generic so that it can be adapted to 
a wide range of approach categories. 


Table 10. Characteristics of our proposal and some context-aware approaches involving HUD 


Approach scope Category HUD Technical HUD involved HUD use Flexibility 
management details 
Adaptation of multimedia Client-side Yes No Context values Multimedia No 
content (movies) to users Multimedia content 
needs in smart homes [33] contents recommendation 
Personalized Server-side No No Learner profile Learning No 
recommendation system Learner model materials 
for learning materials in an recommendation 
ubiquitous learning 
environment [34] 
Context-aware middleware Server-side Yes Yes Context values User interaction No 
supporting reasoning Users actions enhancement 
mechanisms for user 
interaction in cultural 
spaces [40] 
Our proposal is a generic Adaptable to Yes Yes Context values Efficient Yes 
context-aware MDA _ all categories Adaptation multimedia 
architecture supporting (client-side, actions content 
reasoning mechanisms server-side, adaptation 
upon context values and  proxy-based, 
HUD peer-to-peer) 


By relying on hybrid management methods (i.e. combining client-side, server-side, proxy-based, 
and peer-to-peer paradigms) our proposal may provide better data handling because several limitations of 
client-side, proxy-based, and server-side methods can be reduced (e.g. workload balancing, service 
availability) in particular if cloud computing techniques are involved. Thus, by considering the criteria given 
in Table 8, the potential strengths of our approach compared to the works presented in [33], [34], and [40] are 
summarized in Table 11. It should be noted that these values are assigned based on the details provided in the 
corresponding references. Moreover, the difficulty of reconstructing these context-awre systems and the lack 
of several technical details have prevented us from making numerical comparisons with our proposal. 


Table 11. Comparison of our proposal against some context-aware approaches involving HUD 


Ref Security Volume Performance Retrieval Analysis Sharing Scalability 
[33] (2023) Strong Small Average Average Feasible Unsuitable Not scalable 
[34] (2021) Weak Small Average Average Feasible Suitable Not scalable 
[40] (2023) Weak Big data Good Average Feasible Suitable Scalable 
Our proposal Strong Big data Excellent Excellent Easy Suitable Scalable 


In view of the foregoing, we assess our proposal over related-works. The goal is to show the added 
values so as to improve MDA processes. The main advantages of our approach are summarized as follows: 
- The proposal can be considered as a complementary element to existing MDA approaches since most of 
them do not efficiently handle the HUD. 
- The proposal shows adaptability insofar as it handles the HUD at a high level of abstraction, regardless of 
any specific MDA approach (e.g. medical applications, and ubiquitous learning). 
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- The high flexibility as the proposal offers various options to store, retrieve and analyze the HUD. These 
features allow overcoming some of the issues discussed in Table 9 through an efficient placement of the 
HUD manager components in accordance with applications and users needs. 


4.3. Case study on HUD analysis: context-aware adaptation rules learning 
In what follows, we consider a case study to show the usefulness of storing and processing the 

HUD. Thus, the scenarios below give a typical example of how HUD contribute to the personalization of 

adaptation rules through rule learning mechanisms. This task refers to prediction of adaptation actions using 

data-driven approaches (i.e. based on data analysis and interpretation) rather than explicitly defining them. 

- Scenario 1: David is a mobile user. While moving, he accesses multimedia documents on various subjects 
through the mobile data accessible from his smartphone; this is a paid and limited service. To save mobile 
data consumption, he always asks the system to adapt the documents by using low quality media objects 
format. 

- Scenario 2: Sarah is a university student. Back home, she has to revise her online courses using her 
laptop. At that time, Sarah always feels tired due to her busy daily agenda. Therefore, she asks the system 
to adapt the target documents by summarizing their contents. 

These scenarios are implemented in a way that HUD are stored in a NoSQL database, denoted by 

DB, hosted in the proxy. For each scenario, we store two types of HUD. The first type concerns the storage 
of a sufficient number of entries according to the assumptions mentioned in the scenarios. The second type 
consists in storing random entries to endow database DB with dynamicity (i.e. noise) when analysing data. In 
order to infer new personalized adaptation rules, the process given in Figure 7 is proposed. First, the data 
analyser invokes the data retriever to load database DB as an array of objects, denoted by O. Then, array O is 
split into subparts Oj;- 7 ... n according to user id; n is the total number of users. Finally, the elements of each 
subpart Oj=7 ... n, denoted by I={0;j}, are analysed to extract associated conflicts and adaptation actions. 


HUD Manager 
foror D 
MDA process ccc] 
Sensing — : 
Thinking Step 1 f | (Object array 0) Step 2 
Analysis result set 
Data load ~ m Data split 
Request 


data of the i* 
user 


Data 
analyser 
Step 3 


New adaptation rules 


Figure 7. Process phases for adaptation rules learning from HUD 


The idea of the analysis algorithm is to implement an efficient backtracking of associated conflicts 
and actions in each subpart Oj=7.. n, as follows: 

- Let X=(x;, x2, ..., Xm) be a vector such that each element x; E X is bound to one context element; m is the 
number of context elements. The values taken by element x; are defined as all possible qualitative values 
of the corresponding context element in addition to the null value. For example, if x; refers to battery level 
then domain (x)={null, low, medium, full}. 

- Let A={a;, a2, ..., ax} be a set such that each element refers to one of the adaptation actions; k is their 
number. Each element aj=7., E A can be either 0 or 1 to specify whether adaptation action a; is considered 
for execution or not. 

- The backtracking algorithm goes through paths built by combining the possible values for vector X and 
adaptation actions from set A. When the confidence degree of any given combination (x, a) exceeds a 
threshold T, it will be considered as a personalized rule for the i” user. The confidence degree is defined 

as the ratio of (the number of objects from /j={o,} including non-null values from vector x and executed 

actions from vector a) to (the number of objects from J={o0;} including all non-null values from vector x). 


The resulting rules take the form "IF (A ia ,c: © x / ci 4 null) then v;=z.x E a/v; = 1". In addition, each 


time a potential rule is retained, a redundancy check is applied in order to decide whether or not to keep 
this rule. 
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Table 12 shows typical rules retained during the analysis phase. The above scenarios show the 
importance of processing the HUD to build intelligent context-aware applications through adaptation rules 
learning to predict users behaviours. Thus, the main advantages of such a task are summarized as follows: 

- The use of unsupervised machine learning mechanisms; therefore, the context-aware adaptation rules are 
learnt from HUD instead of defining them in a fixed way. Moreover, there will be no need to pre-trained 
data (datasets) to perform the learning task. 

- Possibility to limit the log data so that only some subparts are processed according to a sliding window 
(e.g. considering data corresponding to recent periods). 

- Reusability since it can be adapted to other rule-based context-aware pervasive environments. 

In spite of these advantages, there are still some limitations that should be overcome in future: 

- The system does not involve efficient association rules algorithms (e.g. frequent pattern growth 
algorithm). Indeed, backtracking algorithms cannot be applied to process large volumes of data. That is 
why we need to develop more efficient and effective algorithms for data analysis in order to extract 
personalized adaptation rules. 

- As the system is based on unweighted (blind) correlations, it may take a long time to overlook the most 
frequent old rules and replace them with the most recent ones. 

- The system lacks of experience sharing between users; this feature helps build rule-based 
recommendation systems. 


Table 12. Typical rules retained during the analysis phase 


Rule ID Conflicts Adaptation actions Scenario Confidence (%) 
R1 if (internet data is limited) then Format change Scenario 1 95 
R2 if (night time and busy agenda) then Summarize text Scenario 2 92 


5. CONCLUSION 

The aim of this work has been to involve relational and NoSQL databases to efficiently manage 
HUD resulting from the execution of MDA processes in context-aware pervasive systems. To achieve this 
goal, the context as well as the adaptation actions were first modeled through the oriented-object approach 
and then converted into appropriate schemes. To make the proposal adaptable to several adaptation 
approaches, four management methods have been proposed: client-side management, proxy-based 
management, and server-side management and hybrid management. The proposal has been validated through 
scenarios implemented in a real prototype with a special focus on use cases about context-aware adaptation 
rules learning. In fact, as the aim here was to show only the feasibility of the approach, the developed 
prototype was built around a restricted subset of general adaptation rules independent of any specific field of 
application. For real-world applications, the adaptation rules base may be broadened by considering other 
context elements, depending on the needs of the field of application. In addition, more sophisticated technical 
choices can be involved, such as sensing as service paradigm. Currently, we are working on applying 
machine-learning techniques along with other efficient data analysis algorithms, in order to perform context- 
aware adaptation rules learning from HUD. 
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